• 专利附录
  • 专利说明
  • 权利要求
  • 类似技术
  • 同族专利
  • 引用文献
  • 法律状态
  • 优先权
    • 专利摘要:
    A computer-implemented and computer-controlled method of, and an apparatus and computer program product for clearance of physical content to be inspected, such as container content. The computer collects eye tracking data of a human observer performing the inspection by viewing the content or an image of the content. From real-time processing of the collected eye tracking data, a clearance profile is determined based on expert eye tracking data of visual content inspection, such as container content inspection. The content is cleared by the computer if the determined clearance profile meets a set clearance profile. The eye tracking data may be enhanced by observer performance data collected during the visual inspection.
    公开号:NL2022016A
    申请号:NL2022016
    申请日:2018-11-16
    公开日:2019-05-20
    发明作者:Van Maanen Peter-Paul；Brouwer Anne-Marie；Alrik Zwaan Jonathan
    申请人:Joa Scanning Tech B V；
    IPC主号:
    专利说明:

    Title A computer-controlled method of and apparatus and computer program product for supporting visual clearance of physical content.
    Technical Field
    The present disclosure relates to visual content inspection, and more particularly, to visual content inspection by a computer based on a human observer viewing the content or an image of the content to be inspected, generated by an image generator.
    Background
    Inspection of content, such as container content for security checks, for example the inspection or screening of content of suitcases, hand bags or hand luggage and parcels for transportation by airplanes, trains and busses, and inspection at entrance or access to events, social gatherings, meetings, buildings and or other security checks, for example, is nowadays widely applied an in most cases mandatory by legal requirements.
    This inspection is primarily performed by Transportation Safety Officers (TSOs) and trained special security personnel or agents. It is the task of these officers and agents to identify threat items or potential threat items in containers that often comprise a variety of clutteredly packed items. Besides special detection equipment, such as metal detectors, radiation detectors and explosive detectors, for example, the identification is performed by visual inspection of two-dimensional, 2D, or three-dimensional, 3D, images of the content of the containers generated by an X-ray screening or other scanning technology.
    Container cargo transport by planes, by trucks over land and by ships over sea, for example, for border security and for custom officials to check whether the contents match the cargo description, and whether illegal cargo can be found, is also subjected to visual inspection by an official observer from an image of the content of a container to be inspected.
    By viewing an image of the content, the observer must determine whether a container is believed free from threats or whether a container matches the cargo description, for example, as result of which the container is declared “cleared” by the observer and may pass. Whenever there is a potential threat or suspicion of illegal cargo, for example, the container is subjected to further search, which may involve a time and human resource consuming physical inspection involving the unloading of the container, for example. In case the visual inspection leads to a serious suspicion or threat, the container is even to be handled with extra care and/or inspected under strict safety requirements.
    In practice, if there is an increased risk or if threats are highly prevalent, inspecting officers or agents will more likely make potential or serious threat decisions, such that “clear” responses are slower, because the observer will apply an increased reliability standard in viewing the container content, to avoid as much as possible making a mistake. At airport and access security checks, this may lead to congestions in the security handling and can lead to frustrations with the persons to be cleared and the clearing officials which, eventually, may have a negative influence on the overall quality of the inspection.
    Hand baggage or luggage screening for safety checks, for example, is a repetitive visual search task that often has a very low probability of encountering a threat, but may have high consequences if a serious threat is missed. Since threats are of low prevalence, a “cleared” response becomes the more frequent decision. Observers may tend to abandon a search in less than the average time required to find a target under such circumstances.
    Safety standards applicable to baggage or luggage inspection at airports, for example, where officers or agents have to find a target in the midst of distractors, impose strict operating times on observers, such as shifts of twenty minutes, for example. These operating times are determined, with a high safety margin, from the time during which an average trained human observer can stay focussed. It will be appreciated that, based on these safety requirements, for operating baggage or luggage inspection during the opening hours of an airport, a large team of well-trained security or screening personnel is required.
    Other applications for clearing physical content by visual inspection involve forgery control, such a forged documents like passports, but also paintings and the like, for which a human observer visually inspects the document or painting, but also images of the human or animal body made by security body scan equipment and X-ray equipment, Computed (Axial) Tomography, CT or CAT, scans and Magnetic Resonance Imaging, MRI, in health care applications, for assessing an injury, such as fractures or dislocations, or a disease, for example bone degeneration, infections or tumours, lung and chest problems. In practice, experts like radiologists visually review the pictures and create a report with their findings to aid in diagnosis. In the case of border security control, for example, a customs official checks the identity of a person by viewing the content of a passport, or other ID document, and the person itself.
    With these types of visual inspection, the same problems may arise as outlined above with respect to container content inspection.
    Accordingly, there is need for improving visual inspection of physical content when viewing the content or viewing or monitoring a displayed image of the content by a human observer, for at least one of decreasing the time in which an inspection can be completed by an observer, for increasing consistency and reliability in the decision making process by an observer, and for establishing observer operating and rest times for reliable inspection of the content.
    Summary
    It is an object of the present disclosure to provide a computer-implemented method for visual clearance of physical content to be inspected. It is a further object of the present disclosure to implement such a method in an apparatus for visual clearance of physical content to be inspected, and to provide a container content screening arrangement comprising or arranged for operating with such an apparatus.
    Accordingly, in a first aspect of the present disclosure, there is provided a computer-implemented and computer-controlled method of clearing physical content to be inspected, such as container content, wherein the computer connects to an eye tracking device arranged for providing eye tracking data of a human observer inspecting the content by viewing the content or an image of the content, the computer performing the steps of: collecting eye tracking data of the eye tracking device while the observer viewing the content or the image of the content, real-time processing of the collected eye tracking data for determining a clearance profile based on human expert eye tracking data of visual content inspection, such as container content inspection, and clearing the content if the determined clearance profile meets a set clearance profile.
    Manual clearance of physical content from directly viewing the content itself or from viewing images of the content, requires inspecting officers or agents that are highly trained and experienced in object recognition.
    An image of container content for visual inspection purposes, such as an X-ray image of the container content, for example, generally comprises so-called contoured objects. That is, an object in the container, and the container itself, are depicted by their outer or circumferential contour. Separate parts or items inside an object are also displayed by their contour and, optionally, enhanced by a respective colouring. Hence, the objects inside the container to be inspected, for the greater part, have to be recognized by the observer from their shape or contour and, if available, colour and colour differences. Face recognition, in the case of border security control, for example, requires observers trained in recognizing features of a human face.
    In general, a decision whether to clear a particular physical content is, at present, made in the brain of the observer, from the information and stimuli gained at viewing the content or an image of the content, in correlation with the observer's object and feature knowledge and experience. In this brain process, if the observer does not perceives a dangerous or forbidden object, for example, or a deviation or an abnormality in a document, such a special security element or feature, or in the resemblance of a human object and a picture at an ID document, such that there is no suspicion concerning the viewed content, the observer clears same, i.e. let the content pass.
    Eye tracking technology offers the possibility of capturing visual behaviour in real-time when monitoring or viewing objects and images. In practice, eye tracking technology is proposed for controlling workflows involving a specified sequence of tasks, or for investigating whether all objects in a structured cluster of objects have been viewed at, for example. Eye tracking technology is also proposed for training purposes, for detecting reading or viewing patterns that may lead to poor comprehension, for example. Other applications of eye tracking technology involve monitoring of alertness or distraction and situational awareness of humans while performing a task, such as airplane pilots, truck and car drivers, machine operators, etc.
    Different from applying eye tracking technology in monitoring the performance of a predefined or well specified and structured task, the method according to the present disclosure explicitly makes use of information that can be extracted from the viewing as such of a human observer inspecting unstructured, random or cluttered content, using eye tracking technology.
    According to the present disclosure, clearance of visually inspected content is determined, by the computer, from the visual behaviour of the observer during the inspection, expressed in a clearance profile, by real-time processing of collected eye tracking data of the observer and validating the processed data against expert eye tracking data of visual content inspection. When the determined clearance profile meets a set clearance profile, the content may be automatically cleared by the computer. It has been found that such an automated decision-making process significantly speeds-up content inspection compared to a human decision-making process.
    Instead of massively deploying object or shape recognition technology, requiring high capacity, high resolution and high speed data processing algorithms and equipment, and optionally enhanced by chemical and material sensing technology, for supporting or even replacing a human observer, the present disclosure advantageously makes use of the nowadays readily available and accurate eye tracking technology, producing an amount of data at a rate that allows for real-time processing, not requiring excessive computing equipment, and such that content can be cleared by the computing equipment, even when the human observer in his brain still processes the content information perceived.
    With the present disclosure, the human observer is not replaced and remains crucial. The information as to the decision to be made by the computer is obtained from the visual inspection by the human observer. Visual inspection tasks in the context of the present disclosure are best performed by human observers, which can effectively perceive anomalies within complex scenes, such as content comprised of clutteredly packed items, and that can be readily detected from eye tracking data of the human observer.
    Hence, the disclosed method, so to say, is based on the best of two worlds, data available from the best performers of visual content inspection tasks, i.e. the expert data, and eye tracking data that enable real-time processing while the content inspection is carried out. it will be appreciated that the term real-time processing in the present description and the claims refers to the processing or the execution of data in a short time period after collecting same, providing near-instantaneous output. Real-time data processing is also called stream processing, because of the continuous stream of input data required to timely yield output for the purpose of a process that is momentarily carried out. This in contrast to a batch data processing system, that collects data and then processes all the data in bulk at a later point in time, such that the processing result will become available after all the data have been collected.
    The method according to the present disclosure can be seamlessly-integrated in visual content inspection tasks, either existing or future tasks and systems.
    Hand baggage or luggage applied at airport safety checks, for example, in the majority of cases does not contain suspicious objects or including a threat, hence the majority of these checks is quickly cleared by the computer, before the human observer, thereby effectively reducing the time in which an inspection is completed.
    On the other hand, in case of a threat or suspicion, the particular content is also quickly discriminated as such, i.e. automatically declared not cleared, by the computer and can be subjected to inspection by another observer and/or a special treatment, like physical inspection of a not cleared container, or inspection using special equipment and the like.
    The computer-implemented method according to the present disclosure further provides for an objective inspection, based on the expertise of the human observer, not being prone to whether suspicious content is encountered in a minority or a majority of the inspection tasks, or other external or subjective influences like an increased state of security, for example. The latter can be effectively dealt with in the present disclosure by applying an adaptable clearance profile, i.e. adapting a set clearance profile, or applying a plurality of clearance profiles, i.e. setting a respective clearance profile, that has to be met for clearance of the content.
    In an embodiment of the method according to the present disclosure, the collected eye tracking data at least comprise gaze point data, wherein the clearance profile is determined, by the computer, from at least one of eye fixations, heat maps, areas of interest, time spent, fixation sequences, time-to-first-fixation, saccades, smooth pursuit movements, vergence movements, and vestibule-ocular movements, identified by the computer in the collected eye tracking data.
    Gaze points constitute the basic unit of measure in eye tracking technology. One gaze point equals one raw sample captured by an eye tracking device. The viewing time represented by a gaze point depends on the measurement rate of the eye tracking device. Accordingly, several eye tracking data parameters may be processed from the gaze points.
    If a series of gaze points happens to be close in time and range, the resulting gaze cluster denotes a fixation, a period in which the eyes are locked toward a specific object. A fixation correlates with informative regions of the content under inspection.
    Heat maps are static or dynamic aggregations of gaze points and fixations revealing the distribution of visual attention. Heat maps may also serve as an excellent method to visualize which elements of the stimulus were able to draw attention, using a particular colouring scheme, for example.
    Areas of Interest, AOIs, are user-defined sub-regions of a viewed content. Extracting metrics for separate AOIs provide for evaluating the performance of two or more specific areas in a same image, or document, for example. For example, to compare conditions, or different features within the same content.
    Time spent quantifies the amount of time that observers spent looking at an AOI. As observers have to blend out other stimuli in the visual periphery that could be equally interesting, the amount of time spent often indicates motivation and conscious attention, prolonged visual attention at a certain region of the content clearly points to a high level of interest, while shorter times indicate that other areas of the content or in the environment might be more attractive.
    Fixation sequences are generated based on fixation position and timing information. This is dependent on where observers look and how much time they spend, and provides insight into the order of attention, indicating where observers look first, second, third etc. Fixation sequences also reflect brightness, hue, saturation etc. in a displayed image of the content or environmental content, that are likely to catch attention.
    Time-to-First-Fixation, TTFF, indicates the amount of time it takes an observer to look at a specific area of interest from stimulus onset. TTFF can indicate both bottom-up stimulus driven searches, a risk bearing object catching immediate attention, for example, as well as top-down attention driven searches, wherein observers actively decide to search for certain elements or areas in the content.
    Saccades are rapid eye movements between fixations. Analysis metrics include fixation or gaze durations, saccadic velocities, saccadic amplitudes, and various transition-based parameters between fixations and/or regions of interest. Hence, saccades represent jumps of the eyes while viewing the content. Saccades can be voluntarily, but also occur reflexively, even when the eyes are fixated.
    Smooth pursuit movements are much slower tracking movements of the eye, under voluntarily control. Although smooth pursuit movements are generally made while tracking a moving object or stimulus, highly trained observers can make a smooth pursuit movement in the absence of a moving target. The present disclosure is further not limited to static content, but applies as well to moving physical content.
    Vergence movements align the two eyes of the observer to targets at different distances from the observer. Vergence movements are disjunctive or disconjugate, different from conjugate eye movements in which the two eyes move in the same direction.
    Vestibule-ocular eye movements stabilize the eyes relative to the external world, thus compensating for head movements.
    The clearance profile, in an embodiment, may comprise an observation pattern determined, by the computer, from one or more of the above eye tracking data parameters identified in the collected eye tracking data and patterns of such eye tracking parameters known from or identified in the expert eye tracking data.
    In another embodiment, the clearance profile is determined, by the computer, from anomalies in the above-mentioned processed eye tracking data and parameters identified in the collected eye tracking data compared to the expert eye tracking data, and/or unknown patterns identified in the collected and processed eye tracking data.
    It will be appreciated that a clearance profile may be content dependent, and that several clearance profiles for similar content may be determined and set for providing a clearance.
    In particular when an observer performs the visual inspection by viewing an image of the content, the clearance profile, in an embodiment of the present disclosure, is further determined, by the computer, from at least one of image quality analysis, image dimension analysis and feature extraction analysis of the image of the content.
    Low resolution images, less coloured images, low brightness and low contrast are factors that may alter the viewing behaviour of an observer, for example an abnormal increase of fixation sequences, compared to images of excellent quality, i.e. high resolution, high brightness and contrast, for example. Accordingly, false alarms, i.e. false non-clearance, can be avoided by taking the image quality into account when determining a clearance profile.
    Content with a plurality or very specific Areas of Interest, AOIs, for example, such as special security features that have to be inspected in the content, may require determining the clearance profile by establishing or resolving same, by the computer, for respective parts of the content or respective locations in the image. The eye tracking data obtained and/or the clearance profile obtained may also be normalized against a reference image or reference content, such to rule out human tendencies or preferences for particular parts of the content to be inspected, not particularly contributing to the inspection task.
    In a further advantageous embodiment of the present disclosure, the computer further performs the step of providing annotations based on real-time processing of the collected eye tracking. In particular, the annotating may comprise annotation of a displayed image, such as highlighting of locations at the displayed image.
    Notifications may be provided to the observer while the content is not yet cleared. As the clearance profile is established in real-time, it has been found that deviations in the processed eye tracking data parameters, deviations in the clearance profile itself or deviations obtained while validating same against a set clearance profile, may be used to notify the observer of potential threats, suspicious objects, or otherwise areas of interest, even before the observer him/herself has concluded that a respective area or part of the physical content needs more attention in viewing.
    Notification may be performed by highlighting the image, such as but not limited to hatching, arrowing, encircling, filtering, contrast/brightness changes, filtering out of surrounding image content, colouring, etc. and by providing written instructions, for example. By this type of quick feedback, the viewing behaviour of the observer is effectively supported and a reduced inspection time is achieved, compared to inspection without such support. Notifications may also include audio and/or tactile and/or visual feedback to an observer, such as spoken words through an earpiece or the like and/or vibrations by a vibration mechanism and/or a warning light or lights, for example.
    Notifications may also be provided in case of not yet cleared content, to facilitate physical inspection or another treatment of suspicious content or a so-called second opinion visual inspection, for example. This, to enhance and speed-up as much as possible the screening and clearance of suspicious content. That is, for example, an annotated image of the content to be inspected indicating the suspicious content may be displayed to an officer or other person entrusted with the physical inspection of the not cleared content. It will be appreciated that the computer may be arranged for randomly selecting content for physical inspection, for example for keeping the attention level of an inspector at a required level.
    In a yet further embodiment, clearance of the content is calculated from eye tracking data collected during a set time window starting from presentation of the content or an image of the content to the observer.
    By setting a time window, anomalies in the viewing behaviour of the observer, for example caused by events or distractions in the environment of the observer, fatigue or other lack of alertness, which may have a negative impact on the quality of the inspection, can be efficiently detected. In case of an observer viewing an image of the content to be inspected, the time window may be automatically set by the computer from at least one of image quality analysis, image dimension analysis and feature extraction analysis of the image of the content, and the expert eye tracking data, such to take the into account the quality of an image and/or diversity or variety thereof.
    It will be appreciated that a clearance or a non-clearance by the computer, does not necessarily have to imply that the decision made by the computer should be accepted. The human observer eventually may decide whether to clear, reject, or to elect the content for a further search or investigating, for example by physical inspection of the content of a container, or using additional equipment and tools when inspecting a document, for example.
    Accordingly, in an embodiment of the present disclosure the clearance of the content is further based on receiving, by the computer, of an input of an observer viewing the content or an image of the content to be inspected. The input may be any type of input signal provided by hand, foot, voice, movement of the head, facial expression, or detected from the movement of the eyes of the observer, for example.
    In this embodiment, the final decision to clear the content is made dependent on both the decision of the observer and the decision made by the computer. That is, effectively, the decision whether to clear the content is based on data available from a trained observer and data of experts of visual content inspection tasks, i.e. the expert data used by the computer, resulting in a decision-making process with improved reliability.
    In both cases, that is a clearance decision automatically made by the computer and a clearance decision by both the computer and the observer, the method according to the present disclosure significantly increases the consistency and reliability in the decision making process by an observer inspecting physical content to be cleared. In this context, and for statistical purposes, for example, in a further embodiment of the present disclosure, the computer calculates a figure of merit from the determined clearance profile and the set profile, wherein the content is cleared if the figure of merit is within a set range.
    The eye tracking data obtained and the processed clearance profiles may serve for improving the method, i.e. an algorithm loaded in the computer for processing the eye tracking data and determining clearance profiles, thereby facilitating machine learning.
    In addition to the collection of eye tracking data, the present disclosure also proposes collecting of observer performance data at least one of physiological data, including electroencephalography data, mental data, including at least one of facial expression data, oral expression data and body movements data of the observer while performing the viewing, i.e. the visual inspection, and to determine, by the computer, the clearance profile from real-time processing of both the collected eye tracking data and the observer performance data.
    By associating such observer performance data and the eye tracking data parameters processed from the eye tracking, the quality of the inspection in the sense of an increased number of computer clearances is further improved. The observer performance data may also be used in calculating the figure of merit.
    Among others for establishing observer operating and rest times for reliable inspection of the content, in a further embodiment of the method according to the present disclosure, an inspection quality parameter is determined, by the computer, from cumulative real-time processing of collected data, the inspection quality parameter providing an indicator of the observer performance when viewing the content or an image of the content, wherein if the calculated inspection quality parameter does not meet a set inspection quality level representative of a set inspection quality, the computer providing at least one of feedback information to the observer.
    The inspection quality parameter is based, in an embodiment, on one or more of eye pupil size/dilation, ocular vergence, blinks and distance to the object or image, indicative for body movements.
    An increase in pupil size is referred to as pupil dilation, and a decrease in size is called pupil constriction. Pupil size primarily responds to changes in light (ambient light) or stimulus material (image stimulus). Other properties causing changes in pupil size are emotional arousal (referring to the amount of emotional engagement) and cognitive workload (which refers to how mentally taxing a stimulus is). Pupillary responses may be used as a measure for emotional arousal.
    As explained above, the extraction of vergence, i.e. whether left and right eyes move together or apart from each other, is a natural consequence of focusing of the eye near and far. Divergence often happens when the human mind drifts away, when losing focus or concentration. It can be picked up instantly by measuring inter-pupil distance.
    Eye tracking data can also provide essential information on cognitive workload by monitoring blinks. Cognitively demanding tasks can be associated with delays in blinks, the so-called attentional blink. However, many other insights can be derived from blinks. A very low frequency of blinks, for example, is usually associated with higher levels of concentration. A rather high frequency is indicative of drowsiness and lower levels of focus and concentration.
    Along with pupil size, eye tracking data may also comprise a measurement of the distance to the content or image and the relative position of the observer. Leaning forwards or backwards in front of a remote content is tracked directly and can reflect approach-avoidance behavior.
    The feedback provided if the calculated inspection quality parameter does not meet a set inspection quality level representative of a set inspection quality, can be any of visual, audio or tactile feedback, for example.
    The reliability of a clearance is further enhanced, in an embodiment of the present disclosure, if both the determined clearance profile meets the set clearance profile and the determined inspection quality parameter meets a set inspection quality parameter representative of a set inspection quality.
    In addition to the above-mentioned embodiment, the inspection quality parameter, in a further embodiment, the inspection quality parameter is based on the collecting of at least one of physiological data, including electroencephalography data, mental data, including at least one of facial expression data, oral expression data and body movements data of the observer while performing the visual clearance.
    Another embodiment according to the present disclosure provides that the clearance profile is determined, by the computer, based on collecting and realtime processing of eye tracking data, whether or not in connection with at least one of physiological data, including electroencephalography data, mental data, including at least one of facial expression data, oral expression data and body movements data, or other performance data of a plurality of human observers simultaneously viewing the content or an image of the content to be inspected.
    It has been envisaged that the reliability of the clearance decision by the computer can be further enhanced by collecting and real-time processing of data collected from several observers simultaneously inspecting the same content, such as two or three observers. The observers may view a same image or different images of the same content to be inspected. For example images taken at a different viewing angle.
    The data collected from the individual observers using individual eye tracking equipment and sensors, may be individually processed, in accordance with the method disclosed above, while the decision to clear the content is based on the individual processing results, for example based on a majority decision and/or taking into account the quality of the individual viewings, the experience of the observers, etc.
    Alternatively, the individually collected eye tracking data and/or performance data may be integrally processed by the computer, resulting in a decision whether or not to clear the content.
    The observer or observers may inspect the image of the content at a position remote from where the image of the content is obtained and the number of observers that view an image simultaneously may change over time.
    In a second aspect, the present disclosure provides an apparatus for clearing physical content to be inspected, such as container content, the apparatus comprises a computer and data input/output equipment, the computer being communicatively connected or connectable to a database, the data input equipment comprising at least one eye tracking device arranged for providing eye tracking data of an observer inspecting the content by viewing the content or an image of the content, the computer being arranged for performing the computer-implemented steps of the method disclosed above.
    In a further embodiment, the data input/output equipment of the apparatus further comprises at least one sensor for collecting at least one of physiological data, including electroencephalography data, and mental data, including at least one of facial expression data, oral expression data and body movements data of an observer while viewing the content or image.
    In another embodiment, the data input/output equipment may comprise a display or screen or monitor, for displaying an annotated image of the content to an officer or other person entrusted with the inspection of content that is not cleared and requires further physical inspection.
    The present disclosure, in particular, provides an apparatus or arrangement for container content screening or scanning, comprising a housing having an entrance for receiving containers, such as luggage, baggage or parcels, the content of which having to be inspected by at least one human observer, an exit at which containers that are inspected leaving the apparatus, a transport device for transporting the containers from the entrance to the exit, an image generator, arranged for generating an image of the container content inside the housing for viewing by at last one observer, at least one control console operatively arranged for operating the transport device, and an apparatus or arranged for communicatively connecting to an apparatus according to the second aspect of the present disclosure.
    In an embodiment, the arrangement may comprise a display or monitor at or remote of the exit at which an annotated image of the inspected content is displayed for further physical inspection of content that has not been cleared or for random inspection, for example.
    In a third aspect there is provided a computer program product downloadable from a communication network and/or stored on a computer-readable and/or processor-executable medium, the computer program product comprising program code instructions to cause a computer to carry out the computer-implemented steps of the method according to the present disclosure.
    The above-mentioned and other features and advantages of the present disclosure are illustrated in the following description with reference to the enclosed drawings which are provided by way of illustration only and which are not limitative to the present disclosure.
    Brief Description of the Drawings
    Fig. 1 shows, in a schematic and illustrative manner, an embodiment of an apparatus for clearing physical content.
    Fig. 2 shows, in a schematic and illustrative manner, another embodiment of an apparatus for clearing physical content.
    Figs. 3, 4 and 5 show typical examples of images for visual inspection of physical content, such as screening of a handbag at an airport security check, a truck for transporting cargo, and a scan of human lungs.
    Figs. 6 and 7 show flow chart diagrams of embodiments of the method according to the present disclosure.
    Fig. 8 shows, in a schematic and illustrative manner, based on Fig. 1, an embodiment of an apparatus for clearing physical content operating with a plurality of observers.
    Figs. 9, 10 and 11 show initially obtained results from a pilot test applying the present method, plotted as Receiver Operating Characteristic, ROC, curves.
    Fig. 12 shows, in a schematic and illustrative manner, a luggage or baggage inspection apparatus or arrangement, such as typically used at airport security checks, implementing the teachings of the present disclosure.
    Detailed Description
    It is noted that in the description of the figures, same reference numerals refer to the same or similar components performing a same or an essentially similar function.
    In general, two types of devices for eye tracking are distinguished, the so-called screen-based eye trackers and eye tracking glasses.
    Screen-based type eye trackers, or desktop or stationary eye trackers comprise a monitor or other electronic display device at which an image to be inspected is displayed, such as an image of content to be inspected. At or near the screen eye tracking capturing equipment is positioned, such as one or more digital cameras, for remote eye tracking of a user or an observer that views the image displayed at the screen.
    Eye tracking glasses type eye trackers are devices that are positioned close to the eyes of the observer, usually head-mounted or mounted onto an eyeglass frame. Eye tracking glasses may be used for inspecting content not necessarily displayed on a screen, such as for inspecting hard copy documents, for example.
    Although eye tracking glasses provide observers to move freely when performing a task, compared to screen-based eye trackers providing limited freedom for the observer to move, eye tracking glasses may shift in front of the eyes during data capturing, in particular when they are not mounted correctly.
    Eye-tracking glasses technology may be combined with so-called smart glasses or virtual reality glasses or video headsets, by which an image of content to be inspected is displayed in front of the eyes of an observer. Like screen-based eye trackers, smart glasses eye trackers allow for providing visual feedback or notifications to the user or observer during the inspection, such as by highlighting the image in the form of hatching, arrowing, encircling, filtering, contrast/brightness changes, filtering out of surrounding image content, etc. to support the inspection task and providing written instructions, for example. In case the inspection task is not sufficiently performed, the display of the image may be blocked, for example.
    Figure 1 schematically illustrates an embodiment of an apparatus or assembly 10 for providing computer-implemented clearance of physical content to be inspected, using screen-based eye tracking technology. The apparatus 10 essentially comprises a computer or processor 12 to which an output device connects 36, such as a screen or monitor 11, and an input device connects 35, such as a digital camera or eye tracking capturing module 13. The screen or monitor 11 optionally may comprise an audio output device 14, such as a loudspeaker and/or earpiece, and an audio input device 15, such as microphone, connecting 36 to the computer or processor 12.
    The computer or processor 12 may access at least one database, such as an internal or local data base and/or a local and working memory 16 and/or an external or remote database 18 communicatively connected or connectable to the computer or processor 12 via a data communication network 17, such as the internet. In use, a user or observer 20 views 21 an image 23 of content to be inspected, which image 23 is displayed at the screen or monitor or display 11, while the eyes of the human observer 20 are tracked 19 by the digital camera or eye tracking capturing module 13.
    Reference numeral 22 schematically denotes data input/output equipment of the apparatus 10 comprised of at least one sensor for collecting observer performance data, comprising at least one of physiological data, including electroencephalography data of the observer, while performing the visual clearance. The data input/output equipment 22 connects to the computer 12 by a wired link 34.
    Eye tracking data and sensor data collected may be stored in the database 16, 18 for use by the computer 12.
    Figure 2 schematically illustrates an apparatus or assembly 24 for supporting visual clearance of physical content to be inspected, comprising eye tracking glasses technology. In the embodiment shown, the human user or observer 20 wears an eye glasses device 25 comprised of a frame 28 at which a digital camera or other eye tracking capturing module 27 is fixed.
    In use, the user or observer 20 views 26 a document 30 while eye tracking data 29 captured by the module 27 are transmitted by a wired or wireless transmission link 31 to a computer or processor 12, as explained above with reference to Figure 1. The frame 28, optionally, may comprise a loudspeaker 32 and microphone 33 audio headset, communicatively connected to the computer or processor 12 via the transmission link 31, for example.
    In the embodiment shown, data input/output equipment 22 such as one or more sensor(s) connect by a wireless link 37 to the computer 12. Those skilled in the art will appreciate that in both the embodiments of Figure 1 and Figure 2 the data input/output equipment 22 may connect to the computer 12 by either one of a wired 34 or wireless data communication link 37.
    In the case of a virtual reality glasses or video headsets, the frame 28 comprises a screen or display (not shown) at which an image of the physical content to be inspected is displayed.
    Eye tracking devices as such are readily known and described in the prior art, such that a further description thereof may be omitted here.
    Figure 3 shows a typical example of an X-ray image, available from Wikimedia Commons, obtained with a device for visual inspection of container content such as suitcases, hand bags or hand luggage and parcels for transportation by airplanes, trains and busses, and inspection at entrance or access to events, social gatherings, meetings, buildings and or other security or screening checks, for example.
    As can be seen from the image, objects 41 - 44 in the container or bag 40, and the container or bag 40 itself, are depicted by their outer or circumferential contour. Separate parts or items inside an object, such as the electronic equipment 43 are also displayed by their contour. In practice, these objects may be enhanced by a respective different colouring. The objects 41 - 44 inside the container 40 to be inspected, which are clutteredly packed, i.e. in an unstructured manner, have to be recognized by Transportation Safety Officers, TSOs, and special security personnel or agents by their shape or contour and, if available, colour and colour differences, to identify threat items or potential threat items.
    Figure 4 shows a typical example of an X-ray image, available from Evergreen Shipping Agency (Europe) GmbH Netherlands branch, obtained of a truck 45 and a trailer 46 containing cargo 47, 48 that is normally not visible from the outside, because the cargo 47, 48 is surrounded by a cover. From the image, well trained border security and custom officials may check whether the contents match the cargo description, and whether illegal cargo can be found, for example in randomly positioned hollow spaces in the chassis 49 of the trailer and the truck or between the cargo items 47, 48.
    Figure 5 shows a typical X-ray scan, available from Wikimedia Commons, of the human chest and lungs, from which a trained physician or radiologists by visual inspection of the image 50 has to detect physical content like a tumour or abscess 51, for example.
    In the context of the present disclosure, the term physical content is used to identify real physical objects in the most common meaning thereof, different from virtual or artificial objects.
    In all these inspection examples, by viewing an image of the content, the observer must determine whether the content is believed free from strange, unusual objects or deviations from a normal or healthy state, or matches a particular description, or the like, as result of which the physical content is declared “cleared” by the observer and my pass. The term 'clearance' in the present disclosure is to be interpreted as any state of the content that matches a predetermined condition.
    Although the examples in Figures 3 - 5 show two-dimensional, 2D, images, in practice three-dimensional, 3D, images may be provided for inspection.
    In accordance with the present disclosure, eye tracking data of an observer that is performing a visual content inspection are captured by a computer-controlled arrangement or apparatus 10, 24 as illustrated in Figures 1 and 2, respectively, comprising an eye tracking device such as a screen-based eye tracking device or video headset, in the case of viewing images of the content to be inspected, or an eye glasses based eye tracking device when either one of an image or a real physical object is to be inspected, the latter for example in the case of inspecting original paintings, documents and passport control.
    An example of the method according to the present disclosure is illustrated by the flow chart diagram 60 shown in Figure 6. In the flow chart, the steps proceed from the top to the bottom of the drawing. Other flow directions are indicated by a respective arrow. The illustrated method is implemented and controlled by the computer 12, running a data processing algorithm, suitably programmed by program code instructions that may be stored in the database 16 or loaded therein as a computer program product from a computer readable medium, such as, but not limited from a memory stick, hard-disk, solid-state drive, or other non-transitory medium, or as data signal, downloaded from a data communication network, for example.
    In a first step, content to be inspected is made available for inspection by a human observer, i.e. block 61 "Present content". While the observer views the content or an image of the content, eye tracking data are obtained, i.e. block 62 "Capture eye tracking data", which are real-time processed by the computer 12 for determining a clearance profile, i.e. block 63 "Determining clearance profile".
    It will be appreciated that the clearance profile is determined by the computer 12 from a particular amount of captured eye tracking data, for example during a time period of one, two, three or four seconds, in correlation with previously established expert eye tracking data.
    The amount of expert eye tracking data and the processing algorithm used in determining the clearance profile are content particular, such as a processing algorithm and expert data pertaining to inspection of unstructured or clutteredly packed container content for safety purposes, as illustrated in Figure 3, or a processing algorithm and expert data pertaining to medical content inspection, as illustrated by Figure 5, for face recognition or for forgery content inspection, or a processing algorithm and expert data relating to cargo inspection, as illustrated by Figure 4, for example.
    In decision block 64, i.e. "Meets set profile ", the processing algorithm determines whether the clearance profile established in block 63 meets a set or pre-set clearance profile. In the negative, i.e. decision "No" of block 64, if the clearance profile determined and/or the processed eye tracking data conclude to a suspicious content, such as container content forming a threat, i.e. decision block 65 "Suspicious " answer "Yes", the content may be directly and automatically declared not-cleared by the computer 12, i.e. block 69 "Content not-cleared". If, dependent on the particularities of the determined clearance profile, the computer does not yet directly declare the content not-cleared, i.e. decision "No" of block 65, the computer may decide whether it is appropriate to provide support to the observer, i.e. decision block 66 "Notification ".
    In the affirmative, i.e. decision "Yes" of block 66, the viewing of the observer is supported by one or several notifications or annotations, i.e. block 67, "Notification support". In the negative, i.e. decision "No" of block 66, the capturing of eye tracking data may continue.
    In the embodiment illustrated by the flow chart 60, a time window is introduced, such that clearance of the content is calculated from eye tracking data collected during a set time window starting from presentation of the content in block 61. In decision block 68, i.e. "Time window lapsed ", it is determined whether the time window set by the computer is lapsed. In the negative, i.e. decision "No" of block 68, the processed eye tracking data accumulates and hence the determination of the clearance profile. If the time window is lapsed, i.e. decision "Yes" of block 68, and the content is not already cleared, the content is declared not-cleared, i.e. block 69.
    In the case of content that is not-cleared, further measures may be required, such as a physical inspection of container content, or inspection by a further observer, for example. In all other cases, the content is automatically cleared by the computer 12, merely from processing the captured eye tracking data in relation to the relevant expert eye tracking data, i.e. block 70 "Content cleared".
    The automatic computer-controlled clearance of physical content from the captured eye tracking data processing in correlation with expert data, allows for an improved, i.e. shortened, inspection completion time up till clearance of the content. This is, in particular, advantageous in the case of content inspection for security or screening purposes, such as at airport safety and security inspection desks and entrance desks at events, for example, where large quantities of content have to be inspected with a risk of waiting queues building up.
    It will be appreciated that also in the case of an automatic not-cleared decision by the computer, i.e. block 69 in Figure 6, the inspection process may be advanced, as the computer, from the captured eye tracking data and the expert data, may already quickly after start of the viewing decide to reject the content, such that same can be taken apart and presented for further physical inspection, whether or not enhanced by notifications for an inspector provided by the computer.
    Even in the case of notification support, i.e. block 67 in Figure 6, the inspection completion time till computer clearance of the content can be reduced compared to clearance of the content by the observer without computer control. This, because the notification or annotation can be provided quickly after start of viewing the content. By this fast feedback, the viewing of the content by the observer may be supported such that, in the case of none-suspicious content, a clearance or a noneclearance may be provided, based on the expert data available.
    Notification or annotation support 67 may comprise one or more of highlighting the image 23 at the display 11, for example, such as but not limited to hatching, arrowing, encircling, filtering, contrast/brightness changes, filtering out of surrounding image content, colouring, etc. and by providing written instructions, for example. By way of example, in Figure 3 object 55 is annotated by encircling thereof by a dashed circle 56. This annotation may be shown for a short time, for example, just long enough to control the viewing of the observer towards this object 55, without impeding the view thereof or the view at surrounding or neighbouring objects.
    In the case of a viewing the object itself, such as illustrated in Figure 2, audible notification support may be provided, through the loudspeaker 32, for example, and/or by a light or laser pointer 38, connected 39 and controlled by the computer 12, for example. To facilitate further measures in case of a non-clearance, support may also be provided in the form of the notifications and/or annotations as disclosed above. That is, notifications may also include tactile and/or visual feedback to an observer. To this end, the data input/output equipment 22 may comprise a vibration mechanism and/or a warning light or lights, for example.
    Those skilled in the art will appreciate that certain steps and decisions as shown in the flow chart diagram may be interchanged or differently positioned in the processing flow. For example, decision block 65 may also be positioned between the blocks 63 and 64.
    The time window mentioned above with reference to block 68, may be a fixed time window or dynamically set or adapted by the computer 12. This time window provides an effective means for taking the viewing behaviour of the observer into account. The level of experience of the observer, the size of an image to be inspected and the image quality, determined from a feature extraction analysis of the content, for example, may have an influence on the way the image is viewed. A bad quality, for example, may prolong the viewing time before a decision can be made, or may involve eye movements that are differently valuated compared to a bright and full contrasted image, for example, although the content is such that same may be cleared.
    As mentioned before, an observer may get distracted form viewing by events in his or her environment, by becoming fatigue or other reasons that lead to a lack of alertness which, eventually, may have a negative impact on the quality of the inspection.
    Reference is made to the flow chart diagram 80 shown in Figure 7, illustrating a further embodiment of the present disclosure, implemented and controlled by the computer 12, running a data processing algorithm taking performance data of the observer into account. In block 81 "Present content", the content to be inspected is presented to the human observer 20 (see Figures 1 and 2 and block 61). In addition to eye tracking data of the observer, in block 82 "Capture eye tracking data and performance data", observer performance data are collected, provided by input/output equipment 22 connected to the computer 12, such as a sensor or sensors arranged for collecting at least one of physiological data and electroencephalography data, but also facial expression data of the observer 20 captured by the module 13 or 27, or a separate camera module, such as a video camera module (not shown) and, for example, oral expression data of the observer 20 registered by the microphone 15, 33 and body movements of the observer 20 while performing the visual clearance.
    From the cumulative real time processing of the captured eye tracking data and performance data, both the clearance profile and an inspection quality parameter are calculated, i.e. block 83 "Determine clearance profile and inspection quality parameter". The inspection quality parameter, among others, is representative for the mental and/or physical state of the observer 20 and provides a reliable indicator of the performance of the observer 20 when viewing content 30 or an image 23 of the content at a display or screen or monitor 11.
    The algorithm determines in decision block 84 "Meets set profiles/parameters " whether the clearance profile and/or the inspection quality parameter determined in block 83 meet a set or pre-set clearance profile and quality inspection parameter. In the negative, i.e. decision "No" of block 84, if the clearance profile determined and/or the processed eye tracking data and/or the inspection quality parameter conclude to a suspicious result, either suspicious content and/or a questionable performance by the observer, i.e. decision block 85 "Suspicious " answer "Yes", the content may be directly determined, by the computer 12, as being not-cleared and a notification may be provided indicating a possible reason or pointing to an object that requires further treatment/inspection, for example, i.e. block 89 "Content not-cleared/notification". If, dependent on the particularities of the determined clearance profile/inspection quality parameter, the computer does not yet directly declare the content not-cleared, i.e. decision "No" of block 85, the computer may decide whether it is appropriate to provide support to the observer, i.e. decision block 86 "Notification ".
    In the affirmative, i.e. decision "Yes" of block 86, the viewing of the observer is supported by one or several notifications or annotations and/or performance feedback, i.e. block 87, "Notification support/performance feedback". In the negative, i.e. decision "No" of block 86, the capturing of eye tracking data may continue, dependent on whether a time window has lapsed, i.e. decision block 88 "Time window lapsed ". In the same manner as explained above with reference to Figure 6.
    If the determined inspection quality parameter does not meet a set inspection quality level representative of a required minimum inspection quality, for example, the computer 12 may provide audible, tactile or visible feedback or information to the observer 20 about his/her performance by any of the output equipment disclosed and available with the apparatus 10, 24. Eventually, if the performance gives rise thereto, viewing of the content by the observer 20 may be prohibited, for example.
    In a further embodiment, for benchmarking and for use of the captured data and clearance results for improving the algorithms used, i.e. a self-learning facility of the present disclosure, the computer calculates a figure of merit from the determined clearance profile/inspection quality parameters and said set profile/set inspection quality parameters, as shown in Figure 7 by the dashed block 97 "Figure of merit".
    The figure of merit may also be used for determining whether to clear the content. In that case an additional decision step may be implemented in the flow chart of Figure 7 between the blocks 90 and 91, or the figure of merit may be calculated between the blocks 83 and 84 and may be used in the decision by block 84, for example.
    The clearance decision in blocks 70, 90 and/or the non-clearance decision in blocks 69, 89 may be further based on receiving, by the computer 12, of an input of an observer viewing the content or an image of the content to be inspected. The input may be any type of input signal provided by the input equipment 22, via a touch screen type display 11, a key board or console connecting to the computer 22, etc.
    In this case, the final decision to clear the content is made dependent on both the decision of the observer 20 and the decision made by the computer 12. That is, effectively, the decision whether to clear the content is based on data available from a trained observer 20 and data of experts of visual content inspection tasks, i.e. the expert data used by the computer 12, improving the reliability of the decision.
    The reliability of the computer decision whether or not to clear the content is further increased, in an embodiment of a setting 120 as illustrated in Figure 8, wherein a plurality of human observers 121, 122, 123 simultaneously viewing an image 23 of the content to be inspected.
    With reference to Figure 1, each observer 121, 122, 123 is provided with a screen or monitor 12, and an input device such as a digital camera or eye tracking capturing module 13, connecting by a wired or wireless link for data transfer, such as data bus 35, 36 to a computer or processor 125, which is common to all the observers 121, 122, 123. The computer or processor 125 may access at least one database, such as an internal or local data base or local and working memory 126 and/or an external or remote database 18 communicatively connected or connectable to the computer or processor 125 via a data communication network 17, such as the internet. In use, a user or observer 121, 122, 123 views 21 an image 23 of content to be inspected, which image 23 is displayed at his/her screen or monitor or display 11, while the eyes of the observers 121, 122, 123 are individually tracked 19 by his/her digital camera or eye tracking capturing module 13.
    Each observer 121, 122, 123 may comprise data input/output equipment 22 for individually collecting observer performance data such as comprising at least one of physiological data, including electroencephalography data of the observer, while performing the viewing. The data input/output equipment 22 connect to the computer 125 by a wired or wireless link for data transfer, such as a data bus 34. The individually collected eye tracking data and sensor data may be stored in the database 126, 18 for use by the computer 125.
    It will be appreciated that the observers 121, 122, 123 may also be provided with an apparatus 24 as shown in Figure 2, or a mix of the apparatuses 10 and 24, for example. The number of observers is, of course, not limited to three, and the observers may inspect the image of the content at a position remote from where the image of the content is generated, while the number of observers that view an image simultaneously may change over time.
    The computer 125 is arranged for real-time processing of the data collected from the individual observers 121, 122, 123 while viewing. The data may be individually processed, in accordance with the method disclosed above, while the decision to clear the content is based on the individual processing results, for example based on a majority decision and/or taking into account the quality of the individual viewings, the experience of the observers, etc. as disclosed above. Alternatively, the individually collected eye tracking data and/or performance data may be integrally processed by the computer 12, resulting in a common decision whether or not to clear the content.
    In a pilot verification of the present method, a validation test has been performed involving six experienced and six less experienced human observers, that had to view 200 images of cluttered container content of hand luggage bags or containers, typically presented at entrance security check points, such as at an airport. Of these 200 images, 100 images where completely free of threat or suspicious items, and 100 images contained at least one threat item or potential threat or suspicious item. The test was performed using collected eye-tracking data only and results of the present method were compared against the decision of the human observer, i.e. which results comprised of the threat items or potential threats or suspicious items as indicated by the human observers.
    The viewing time of the individual observers has been represented by gaze point data, resulting in gaze point clusters, denoting a fixation of the viewing at an informative region of the content under inspection in a particular time window of, for example, one or two, three, four and up to 7 - 10 seconds, a period in which the eyes are locked towards a specific object. Based on the gaze clusters, the computer made a decision whether the content was to be cleared or not, i.e. had to presented for further inspection.
    The initially obtained results are illustrated in so-called Receiver Operating Characteristic, ROC, curves, shown in Figures 9, 10 and 11, in which the True Positive Rate, TPR, indicated at the vertical axis, is plotted against the False Positive Rate, FPR, indicated at the horizontal axis, for various decision threshold settings. An ROC curve is a commonly used way to visualize the performance of a binary classifier, meaning a classifier with two possible output classes.
    In the curves of Figures 9, 10 and 11, the TPR defines how often the computer predicts positive, i.e. a possible threat, i.e. meaning content that should be physically inspected, when the actual classification is positive, i.e. the number of true positives compared to the number of all positives, according to the results of the observers. The FPR defines when the actual classification is negative, i.e. no threat, meaning content to be cleared or passed, how often the computer incorrectly predicts positive, i.e. the number of false positives compared to the number of all negatives, according to the results of the observers.
    The top-right position of the decision threshold line refers to a threshold setting of 0, and the bottom-left position refers to a threshold setting of 1. Accuracy is measured by the Area Under the ROC Curve, AUC. An area of 1 represents a perfect test and an area of 0.5 represents a worthless test, i.e. with a performance of a random classifier.
    Figure 9 illustrates the ROC curve of the decisions made by the computer based on all gaze data of the human observers obtained in a viewing time window of 7 seconds. As can be seen from this curve, a 94% hit rate is achieved, i.e. when the observers are also of the opinion that a threat exists, against an only 3% false alarm rate, i.e. the operators are also of the opinion that there is no threat. With a very good AUC of 0.9796.
    Figure 10 illustrates the ROC curve of the decisions made by the computer after a viewing window of 2 seconds. The hit rate is already at 77% and the false alarm rate is only at 2%, with AUC = 0.8317. Figure 11 illustrates the ROC curve within 1 second, achieving already a hit rate of 70% and only a 2% false alarm rate, with AUC = 0.7564.
    Hence, from this pilot experiment, already after 1 second of viewing by a trained human observer, with the method presented, the computer can already find 70% of the possible threats, according to the result of a trained observer, while 98% of the items containing no threat are not falsely classified as a threat. Taken into account that a trained observer makes a decision after about 7-10 seconds, the pilot results demonstrate the ability of the present method for speeding-up and improving a content screening process.
    Figure 12 shows a typical application of the present screening method in a luggage or baggage inspection or screening apparatus or arrangement 100. The apparatus or arrangement 100 comprises at one side an entrance 101 for receiving luggage or parcels or containers 112, the content of which has to be inspected and at another opposite side an exit 102 at which luggage or containers 113 that are inspected leave the apparatus or arrangement 100. The luggage or containers are typically transported through the apparatus or arrangement 100 by a transport belt 103. The belt 103 is automatically stopped when a bag or other container has entered the apparatus or arrangement 100 for inspection.
    At this stage an image, typical an X-ray image, is generated from the content of the container inside the apparatus or arrangement 100 by an image generator 107, positioned inside the apparatus 100. At an operator side of the apparatus or arrangement 100, a screen or monitor or display 104 and a control console 105 is positioned, for use by an operator or observer performing the inspection. For capturing eye tracking data of the observer, an eye tracking module 108 is provided in the vicinity of the display 104. It will be appreciated that the display 104, control console 105, and eye tracking module 108 may be arranged remote from the apparatus or arrangement 100. In that case, the apparatus or arrangement 100 is provided with data transfer equipment for communicatively exchanging image data and the like.
    The apparatus or arrangement 100, inclusive the belt 103, image generator 107 and display 104 are operated under control of a computer or processor 106, for example positioned inside the apparatus or arrangement 100. The setup of the computer 106 and display 104 is equivalent to the equipment shown in Figure 1, i.e. computer 12, data bases 16, 18, data communication network 17, eye tracking module 13, and the input/output devices 14, 15 and 22, although the latter are not explicitly shown in Figure 8 as same may be omitted in the simplest embodiment of the present disclosure. It will be appreciated that the setup 24 of Figure 2 may also be used.
    The apparatus or arrangement 100, in practice, may form part of a security or screening system, involving further screening devices and/or further observers, such as described with reference to Figure 8.
    For facilitating physical inspection of content that is not cleared as a pass, a further screen or monitor or display 109 and a control console 110, such as keyboard, is provided at or remote of the exit 102, communicatively connecting 111 to the computer 106 for receiving an annotated image of the content selected for physical inspection.
    For real-time processing of the captured eye tracking data of the observer and/or the observer performance data, several processing algorithms known in practice may be used, for example based on Markov chains, determining one or a plurality of eye fixations, heat maps, areas of interest, time spent, fixation sequences, time-to-first-fixation, saccades, smooth pursuit movements, vergence movements, and vestibule-ocular movements, as elucidated in the Summary part above.
    The clearance profile, in an embodiment, comprises an observation pattern determined, by the computer, from the collected and processed eye tracking data and known patterns identified from the expert eye tracking data, whether or not enhanced by collected observer performance data. In another embodiment of the present disclosure, the clearance profile is determined, by the computer, from anomalies identified in the collected eye tracking data compared to the expert eye tracking data and/or unknown patterns identified in the collected and processed eye tracking data, whether or not enhanced by collected observer performance data.
    In an embodiment, the clearance profile is determined, by the computer, from a function or model comprising a plurality of expert function or model parameters derived from a collection of observer data gathered from well trained, experienced observers, i.e. experts, while performing a specific visual content inspection task. The data gathered at least comprises eye tracking data and, optionally, observer performance data as elucidated above. Such a function may be established by data analysis experts and/or processor or processing machine controlled using neural network technology, for example.
    From the captured eye tracking data and, optionally, collected observer performance data while performing the visual inspection task, function or model parameters in accordance with a task specific function or model are determined by the computer and correlated with the expert function or model parameters. Thereby determining a clearance profile for the purpose of computer-controlled clearance of the inspected content in accordance with the present disclosure.
    Algorithms for calculating an inspection quality parameter from collected performance data of the observer may be developed and applied in a like manner as explained above with respect to the clearance profile, for example.
    Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measured cannot be used to advantage. Any reference signs in the claims should not be construed as limiting the scope thereof.
    权利要求:
    Claims (15)
    [1]
    A computer-implemented and computer-controlled method for releasing physical content to be inspected, such as container content, wherein the computer is connected to an eye-tracking device adapted to provide eye-tracking data from an observer who inspects the content by viewing the content or a image content, which computer performs the steps of: collecting eye-tracking data from the eye-tracking device while the observer is viewing the content or image, real-time processing of the collected eye-tracking data to determine a release profile based on expert eye-tracking data from visual content inspection, such as container content inspection, and releasing the content if the determined release profile meets a set release profile.
    [2]
    The method of claim 1, wherein the collected eye tracking data comprises at least gaze point data, wherein the release profile is determined by the computer from at least one of eye fixations, heat maps, areas of interest, time spent, fixation sequences, time to first fixation, saccades, smooth pursuit movements, vergence movements, and vestibule-ocular movements, identified by the computer in the collected eye-tracking data.
    [3]
    The method of any one of the preceding claims, wherein the observer views an image of the content and the release profile is further determined by the computer from at least one of image quality analysis, image dimension analysis, and feature extraction analysis of the image.
    [4]
    Method according to any of the preceding claims, wherein the release profile is further determined by the computer from collecting and real-time processing of at least one of physiological data, including electroencephalography data, mental data, including at least one of facial expression data, oral expression data and physical movement data of the observer during viewing.
    [5]
    A method according to any one of the preceding claims, wherein the computer further performs the annotation step based on the real-time processing of the collected data, in particular wherein the annotation comprises marking locations in an image of the content .
    [6]
    The method of any one of the preceding claims, wherein release of the content is based on the data collected by the computer during a set time window starting from the presentation of the content or the representation of the content to the observer.
    [7]
    The method of any one of the preceding claims, wherein the release of the content is further based on receiving, by the computer, an input from an observer viewing the content or the image of the content.
    [8]
    A method according to any one of the preceding claims, wherein the release profile is an observation pattern determined by the computer from one or more of eye-tracking parameters identified in the collected eye-tracking data and patterns of these eye-tracking parameters known or identified from the expert eye-tracking data. includes.
    [9]
    A method according to any one of the preceding claims, wherein the release profile is determined by the computer from at least defects identified in the collected eye-tracking data compared to the expert eye-tracking data.
    [10]
    A method according to any one of the preceding claims, wherein the computer calculates a quality factor from the determined release profile and the set release profile, the content being released if the quality factor is within a set range, in particular wherein the computer calculates from cumulative realization. time processing of collected data an inspection quality parameter is determined, which inspection quality parameter provides an indicator of the performance of the observer during viewing, wherein if the calculated inspection quality parameter does not meet a set inspection quality level representative of a set inspection quality, the computer at least provides feedback information to the observer and wherein the content is released if both the determined release profile meets the set release profile and the determined inspection quality parameter meets a set inspection quality parameter repr optional for a set inspection quality.
    [11]
    A method according to any one of the preceding claims, wherein the release profile is determined by the computer based on the collection and real-time processing of eye-tracking data, whether or not in conjunction with at least one of physiological data, including electroencephalogography data, mental data, comprising at least one of facial expression data, oral expression data, and body movement data, from a plurality of observers simultaneously viewing the content to be inspected or an image of the content to be inspected.
    [12]
    12. Device for releasing physical content to be inspected, such as container content, which device comprises a computer and data input / output equipment, which computer is communicatively connected or connectable to a database, which data input equipment comprises at least one eye-tracking device adapted to providing eye-tracking data from an observer inspecting the content by viewing the content or an image of the content, which computer is adapted to perform the computer-implemented steps according to any of claims 1-11.
    [13]
    The apparatus of claim 12, wherein the data input / output equipment further comprises at least one sensor for collecting at least one of physiological data, including electroencephalography data, and mental data, including at least one of facial expression data, oral expression data and body motion data of an observer during viewing.
    [14]
    14. Container content screening device, comprising a housing with an entrance for receiving containers, such as travel goods, luggage or packages, the contents of which must be inspected by at least one observer, an exit where inspected containers leave the device, a transport device for transporting the containers to the exit from the containers, an image generator adapted to generate an image of the container contents within the housing for viewing by at least one observer, and a device or device adapted to communicate with a device according to claim 12 or 13.
    [15]
    A computer program product that can be loaded from a communication network and / or stored on a computer readable and / or computer processable medium, which computer program product comprises program code instructions to cause a computer to perform the computer implemented steps of any one of claims 1 -11.
    类似技术:
    公开号 | 公开日 | 专利标题
    US20110043759A1|2011-02-24|Method and system for determining familiarity with stimuli
    CN107133584A|2017-09-05|Implicit intention assessment sorting technique based on eye-tracking
    Dowd et al.2013|Attentional guidance by working memory overrides salience cues in visual search.
    Marchi et al.2019|Automated pose estimation captures key aspects of General Movements at eight to 17 weeks from conventional videos
    Graves et al.2011|The role of the human operator in image-based airport security technologies
    Inoue et al.2013|Automated microaneurysm detection method based on eigenvalue analysis using hessian matrix in retinal fundus images
    CN111343927A|2020-06-26|Cognitive dysfunction diagnosis device and cognitive dysfunction diagnosis program
    Long et al.2017|A familiar-size Stroop effect in the absence of basic-level recognition
    US20180314327A1|2018-11-01|System and methods for evaluating images and other subjects
    Lancry-Dayan et al.2018|Do you know him? Gaze dynamics toward familiar faces on a concealed information test
    Adhikari et al.2017|Video‐based eye tracking for neuropsychiatric assessment
    Foulsham et al.2011|If visual saliency predicts search, then why? Evidence from normal and gaze-contingent search tasks in natural scenes
    Gibadullin et al.2020|Mobile Application for Neural Network Analysis of Human Functional State
    Karchani et al.2015|The Steps of proposed drowsiness detection system design based on image processing in simulator driving
    NL2022016B1|2021-05-31|A computer-controlled method of and apparatus and computer program product for supporting visual clearance of physical content.
    CN109620266A|2019-04-16|The detection method and system of individual anxiety level
    WO2018029361A1|2018-02-15|Improving the quality and power of baggage inspection systems
    Piasecki et al.2017|Improving Anomaly Detection Through Identification of Physiological Signatures of Unconscious Awareness
    Botzer et al.2015|Effects of cues on target search behavior.
    Doll et al.1998|Robust, sensor-independent target detection and recognition based on computational models of human vision
    Xu et al.2019|Ultra-rapid object categorization in real-world scenes with top-down manipulations
    Zappalà et al.2013|Differentiating sexual preference in men: Using dual task rapid serial visual presentation task
    Hale et al.2008|Using physiological measures to discriminate signal detection outcome during imagery analysis
    Balas et al.2015|Face and body emotion recognition depend on different orientation sub-bands
    Bruer et al.2017|A repeated forced‐choice line‐up procedure provides suspect bias information with no cost to accuracy for older children and adults
    同族专利:
    公开号 | 公开日
    WO2019098835A1|2019-05-23|
    NL2022016B1|2021-05-31|
    NL2019927B1|2019-05-22|
    引用文献:
    公开号 | 申请日 | 公开日 | 申请人 | 专利标题
    GB0122601D0|2001-09-19|2001-11-07|Imp College Innovations Ltd|Manipulation of image data|
    US8020993B1|2006-01-30|2011-09-20|Fram Evan K|Viewing verification systems|
    ITFI20120072A1|2012-04-05|2013-10-06|Sr Labs S R L|METHOD AND SYSTEM FOR IMPROVING THE VISUAL EXPLORATION OF AN IMAGE DURING THE SEARCH FOR A TARGET.|
    法律状态:
    优先权:
    申请号 | 申请日 | 专利标题
    NL2019927A|NL2019927B1|2017-11-16|2017-11-16|A computer controlled method of and apparatus and computer program product for supporting visual clearance of physical content.|
    [返回顶部]